Cognitive virtual detector

ABSTRACT

Aspects of the present invention disclose a method, computer program product, and system for detecting and mitigating adversarial virtual interactions. The method includes one or more processors detecting a user communication that is interacting with a virtual agent. The method further includes one or more processors determining a risk level associated with the detected user communication based on one or more actions performed by the detected user while interacting with the virtual agent. The method further includes one or more processors in response to determining that the determined risk level associated with the detected user communication exceeds a risk level threshold, initiating, a mitigation protocol on interactions between the detected user and the virtual agent, where the mitigation protocol is based on the actions performed by the detected user while interacting with the virtual agent.

BACKGROUND

The present invention relates generally to the field of cybernetics, andmore particularly to artificial intelligence.

In artificial intelligence, an intelligent agent (IA) is an autonomousentity which observes through sensors and acts upon an environment usingactuators (i.e., it is an agent) and directs its activity towardsachieving goals (i.e., it is “rational,” as defined in economics).Intelligent agents may also learn or use knowledge to achieve theirgoals. They may be very simple or very complex: a reflex machine, suchas a thermostat, is an intelligent agent.

Simple reflex agents act only on the basis of the current percept,ignoring the rest of the percept history. The agent function is based onthe condition-action rule: if condition then action. This agent functiononly succeeds when the environment is fully observable. Some reflexagents can also contain information on their current state which allowsthem to disregard conditions whose actuators are already triggered.Infinite loops are often unavoidable for simple reflex agents operatingin partially observable environments.

A model-based agent can handle partially observable environments. Itscurrent state is stored inside the agent maintaining some kind ofstructure which describes the part of the world which cannot be seen.This knowledge about “how the world works” is called a model of theworld, hence the name “model-based agent.” A model-based reflex agentshould maintain some sort of internal model that depends on the percepthistory and thereby reflects at least some of the unobserved aspects ofthe current state. Percept history and impact of action on theenvironment can be determined by using internal model. It then choosesan action in the same way as reflex agent.

Goal-based agents further expand on the capabilities of the model-basedagents by using “goal” information. Goal information describessituations that are desirable. This allows the agent a way to chooseamong multiple possibilities, selecting the one which reaches a goalstate. Search and planning are the subfields of artificial intelligencedevoted to finding action sequences that achieve the agent's goals.Goal-based agents are more flexible because the knowledge that supportsits decisions is represented explicitly and can be modified.

Goal-based agents only distinguish between goal states and non-goalstates. It is possible to define a measure of how desirable a particularstate is. This measure can be obtained through the use of a utilityfunction which maps a state to a measure of the utility of the state. Amore general performance measure should allow a comparison of differentworld states according to exactly how happy they would make the agent.The term utility can be used to describe how “happy” the agent is. Arational utility-based agent chooses the action that maximizes theexpected utility of the action outcomes—that is, what the agent expectsto derive, on average, given the probabilities and utilities of eachoutcome. A utility-based agent has to model and keep track of itsenvironment, tasks that have involved a great deal of research onperception, representation, reasoning, and learning.

Learning has the advantage that it allows the agents to initiallyoperate in unknown environments and to become more competent than itsinitial knowledge alone might allow. The most important distinction isbetween the “learning element,” which is responsible for makingimprovements, and the “performance element,” which is responsible forselecting external actions. The learning element uses feedback from the“critic” on how the agent is doing and determines how the performanceelement should be modified to do better in the future. The performanceelement is what we have previously considered to be the entire agent: ittakes in percepts and decides on actions. The last component of thelearning agent is the “problem generator.” It is responsible forsuggesting actions that will lead to new and informative experiences.

SUMMARY

According to one embodiment of the present invention, a method fordetecting and mitigating adversarial virtual interactions is provided.The method for detecting and mitigating adversarial virtual interactionsmay include one or more processors detecting a user communication thatis interacting with a virtual agent. The method further includes one ormore processors determining a risk level associated with the detecteduser communication based on one or more actions performed by thedetected user while interacting with the virtual agent. The methodfurther includes one or more processors in response to determining thatthe determined risk level associated with the detected usercommunication exceeds a risk level threshold, initiating, a mitigationprotocol on interactions between the detected user and the virtualagent, wherein the mitigation protocol is based on the actions performedby the detected user while interacting with the virtual agent.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a distributed dataprocessing environment, in accordance with an embodiment of the presentinvention.

FIG. 2 is a flowchart depicting operational steps of a program fordetecting and mitigating adversarial conversations with virtual agents,in accordance with an embodiment of the present invention.

FIG. 3 illustrates an example of a program for detecting and mitigatingadversarial conversations with virtual agents, in accordance with anembodiment of the present invention.

FIG. 4 is a block diagram of components of a computer system, such asthe server computer of FIG. 1, in accordance with an embodiment of thepresent invention.

DETAILED DESCRIPTION

Embodiments of the present invention recognize that virtual agents(e.g., virtual intelligent agents) are increasingly being deployed inenterprises to handle interactions with customers or with employees.Virtual agents are increasingly becoming targets for attack and abuse,for example, by bot generated spam traffic to saturate bandwidth ordrive up operating costs. Back end proprietary virtual agent models arevulnerable to extraction attacks for the purpose reverse engineeringproprietary model functionality or to extract proprietary informationfrom training data. Virtual agents that continuously learn fromproduction use are targets for poisoning attacks. In this scenario,shifting distributions in the training data are employed to drive theconversations off course. Virtual agents used in command controlscenarios are targets for attack by the attacker learning and exploitingweak spots in the underlying models to mislead or fool the virtualagent. An example of this is covertly taking over control of the phoneof a user via hidden voice commands to a speech control chatbot.

Embodiments of the present invention recognize that current approachesto detect attacks on virtual agents are not adequate to detect semanticapplication level attacks, such as extraction, poisoning, and evasionattacks, as well as more sophisticated spam attacks that model humanbehavior and are conducted at a lower volume.

Embodiments of the present invention provide a system built alongsidethe virtual agent that supervises the work of the virtual agent using amethod that employs three subsystems. Embodiments of the presentinvention provide a detection subsystem to analyze utterances and detectsuspicious user behavior. The system includes an ensemble of detectionmodels to examine user behavior. Embodiments of the present inventionprovide a deception subsystem to redirect virtual agent responses upon araised level of suspicion. Embodiments of the present invention providea probing subsystem to maximize the learning about the user throughinformation gathering probes. Hidden dialogue is injected in thedialogue flow with the goal of revealing adversarial user intent.

Example embodiments, in accordance with the present invention, will nowbe described in detail with reference to the Figures. FIG. 1 is afunctional block diagram, illustrating distributed data processingenvironment 100. Distributed data processing environment 100 includescomputing device 110 and server 120, interconnected over network 185.

In one embodiment, computing device 110 includes graphical userinterface (GUI) 130, web browser 150, and storage 160. The variousprograms on computing device 110 include a web browser, an electronicmail client, security software (e.g., a firewall program, a geo-locatingprogram, an encryption program, etc.), an instant messaging (IM)application (app), and a communication (e.g., phone) application.

Computing device 110 may be a desktop computer, a laptop computer, atablet computer, a specialized computer server, a smartphone, a wearabledevice (e.g., smart watch, personal fitness device, personal safetydevice), or any programmable computer system known in the art with aninteractive display or any other computer system known in the art. Incertain embodiments, computing device 110 represents a computer systemutilizing clustered computers and components that act as a single poolof seamless resources when accessed through network 185, as is common indata centers and with cloud computing applications. In general,computing device 110 is representative of any programmable electronicdevice or combination of programmable electronic devices capable ofexecuting machine-readable program instructions and communicating withother computer devices via a network.

In one embodiment, graphical user interface 130 operates on computingdevice 110. In another embodiment, graphical user interface 130 operateson another computer in a server-based setting; for example, on a servercomputer (e.g., server 120). In yet another embodiment, graphical userinterface 130 operates on computing device 110 simultaneously with aserver computer interconnected through network 185 (e.g., server 120).Graphical user interface 130 may be any user interface used to accessinformation from computing device 110, such as information gathered orproduced by program 200. Additionally, graphical user interface 130 maybe any user interface used to supply information to computing device110, such as information supplied by a user for input to program 200. Insome embodiments, graphical user interface 130 may present a generic webbrowser used to retrieve, present, and negotiate resources from theInternet. In other embodiments, graphical user interface 130 may be asoftware or application that enables a user at computing device 110access to network 185.

In yet another embodiment, a user of computing device 110 can interactwith graphical user interface 130 through a touch screen that performsas both an input device to a graphical user interface (GUI) and as anoutput device (i.e., an electronic display) presenting a plurality oficons associated with software applications or images depicting theexecuting software application. Optionally, a software application(e.g., a web browser) can generate graphical user interface 130operating within the GUI of computing device 110. Graphical userinterface 130 accepts input from a plurality of input/output (I/O)devices including, but not limited to, a tactile sensor interface (e.g.,a touch screen or a touchpad) referred to as a multi-touch display. AnI/O device interfacing with graphical user interface 130 may beconnected to computing device 110, which may operate utilizing wired(e.g., USB port) or wireless network communications (e.g., infrared,NFC, etc.). Computing device 110 may include components, as depicted anddescribed in further detail with respect to FIG. 4, in accordance withembodiments of the present invention.

Web browser 150 may be a generic web browser used to retrieve, present,and traverse information resources from the Internet. In someembodiments, web browser 150 may be a web browser designed for a mobiledevice. In other embodiments, web browser 150 may be a web browserdesigned for a traditional computing device, such as a desktop computer,PC, or laptop. In general, web browser 150 may be any application orsoftware that enables a user of computing device 110 to access a webpageover network 185. In the depicted environment, web browser 150 resideson computing device 110. In other embodiments, web browser 150, orsimilar web browsers, may reside on other computing devices capable ofaccessing a webpage over network 185.

Storage 160 (e.g., a database) located on computing device 110,represents any type of storage device capable of storing data that isaccessed and utilized by computing device 110. In other embodiments,storage 160 represents multiple storage devices within computing device110. Storage 160 stores information such as, but not limited to, accountinformation, credentials for authentication, user preferences, lists ofpreferred users, previously visited websites, history of visited Wi-Fiportals, and the history of the location of the computing device.

In general, network 185 can be any combination of connections andprotocols that will support communications among computing device 110.Network 185 can include, for example, a local area network (LAN), a widearea network (WAN), such as the Internet, a cellular network, or anycombination of the preceding, and can further include wired, wireless,and/or fiber optic connections.

Server 120 may be a desktop computer, a laptop computer, a tabletcomputer, a specialized computer server, a smartphone, or any othercomputer system known in the art. In certain embodiments, server 120represents a computer system utilizing clustered computers andcomponents that act as a single pool of seamless resources when accessedthrough network 185, as is common in data centers and with cloudcomputing applications. In general, server 120 is representative of anyprogrammable electronic device or combination of programmable electronicdevices capable of executing machine-readable program instructions andcommunicating with other computer devices via a network. In oneembodiment, server 120 includes database 170 and program 200.

In an embodiment, server 120 is capable of initiating a handshakeprocess between server 120 and computing device 110. Handshaking is anautomated process of negotiation that dynamically sets parameters of acommunications channel established between two entities before normalcommunication over the channel begins. Handshaking follows the physicalestablishment of the channel and precedes normal information transfer.Handshaking facilitates connecting heterogeneous computing systems, orequipment, over a communication channel without the need for userintervention to set parameters. In an example, server 120 initiates thehandshake process by sending a massage to computing device 110indicating that server 120 wants to establish a communication channel inorder to gain access to programs on computing device 110.

Database 170 may be a repository that may be read by server 120.Database 170 represents any type of storage device capable of storingdata that is accessed and utilized by server 120. In other embodiments,database 170 represents multiple storage devices within server 120.Database 170 stores information such as, but not limited to, accountinformation, credentials for authentication, user preferences, lists ofpreferred users, previously visited websites, history of visited Wi-Fiportals, and the history of the computing devices, and informationlocated on the computing devices, that access the server.

Virtual agent program 175 is a program on server 120. In an embodiment,virtual agent program 175 is an animated, artificial intelligencevirtual character with anthropomorphic appearance that serves as anonline customer service representative. In an example, virtual agent 175already understands many of the conversations that a user wants to havebecause it comes pre-trained with industry and domain content. VirtualAgent program 175 applies cognitive technology to provide apersonalized, contextualized customer experience, with pre-trainedindustry and domain knowledge. In another embodiment, virtual agentprogram 175 leads an intelligent conversation with users, responds touser questions and performs adequate non-verbal behavior. Virtual agentprogram 175 is capable of using engagement metrics to understand moreabout the conversations that virtual agent program 175 is having withone or more users. In another embodiment, virtual agent 175 may alsoappear as holographic projections welcoming customers in a hotel lobby,restaurant, or business office reception.

Detector 180 is a sub program of program 200 that intercepts requestsissued by users to, and responses by, virtual agent program 175 andfunctions as a dialogue anomaly detection sub system of program 200. Inan embodiment, detector 180 extracts the log entry requests andresponses by virtual agent program 175, and detector 180 uses the logentry as input to generate an ensemble of anomaly detection models. Inthis embodiment, detector 180 is capable of generating a detection modelthat is unique to a different detection strategy. In an example,detector 180 generates and merges each of the following detectionmodels: a Markov model to analyze natural language, an informationleakage model to track the distribution of queries to the model anddetermine risky levels of information exposure, a timing model thatinspects time tags to detect suspiciously fast answers for a human, aconfidence model that flags the occurrence of low confidence scores onrecognized intents, and a dialog progression model that identifies theabsence or presence of dialogue progress indicators such as theoccurrence of certain target states in the dialogue with a user.Detector 180 combines the individual detection models into a single riskscore that is weighted to assist in determining the suspiciousness ofthe response.

Bot shield database 182 is a database located on program 200 and usedexclusively by program 200. Bot shield database 182 represents any typeof storage device capable of storing data that is accessed and utilizedby program 200. In other embodiments, bot shield database 182 representsmultiple storage devices within program 200. Bot shield database 182stores information such as, but not limited to, context information,account information, credentials for authentication, user preferences,and lists of preferred users. For example, bot shield database 182stores words and phrases that are associated with a high-risk levelsuggesting that the words are associated with suspicious activity. Botshield database 182 stores a history of data from individual detectionmodels. For example, bot shield database 182 stores one or morehistories of timing response patterns to a timing detection model thatare associated with high risk, medium risk, and low risk users. Inanother example, bot shield database 182 stores information from aMarkov detection model. In this example, bot shield database 182 storeshistorical information, flagged as inappropriate, of unlikelyinteractions between a virtual agent and a user. Bot shield database 182stores the history of the transition frequencies from the dialog statesbetween a user and a virtual agent.

In another embodiment, bot shield database 182 stores historicalinformation of the effectiveness in detecting anomalies for individualanomaly detection models, and the anomaly detection models ensemble inone or more arrangements. In another embodiment, bot shield database 182stores historical information of extensions and updates of additionaldetection models in the anomaly detection subsystem, mitigationresponses in the mitigation system, and probe choices in the probingsystem. In an example, program 200 receives updates of new evasivemalware and upgraded defenses capable of successfully mitigating thenovel attack to a virtual agent. Bot shield database 182 stores theinformation of new evasive malware, and the upgraded defenses to themalware.

Deception engine 190 is a sub program of program 200 that automaticallyadjusts the fidelity of a response, to a user, by virtual agent program175 to deter a potential attack. Deception engine 190 mitigates attacksby changing the fidelity, or exactness, of model responses withoutaltering original dialog flow with a user. Deception engine 190 changesthe exactness of the model responses given to a user by selecting afidelity level of response according to the current user risk scores.The higher the user risk, the lower the precision of the model responsegiven to the high-risk user. For example, if the model response, priorto deception engine 190 changing the fidelity of the response, is“please confirm that you entered credit card number ###-####-####” thendeception engine 190 in response to high-risk activity by a user,changes the fidelity of the response consistent with the risk level ofthe user. In this example, deception engine 190 changes the fidelity ofthe model response to “please re-enter your credit card number toconfirm.” Deception engine 190 is triggered if the user risk scorepasses a certain predefined threshold. In an embodiment, deceptionengine 190 triggers mitigation actions based on the specific risk levelof a user and the fact that a specific threshold was passed based on theuser responses to virtual agent program 175.

In another embodiment, deception engine 190 is capable of mitigatingattacks on virtual agent program 175, in addition to redirectingdialogue flow to safe dialogue as previously discussed by changing thefidelity of model responses. In an embodiment, deception engine 190 usesa plurality of strategies to create lower fidelity responses, such asprogressive model dilution. In an example, deception engine 190 usespreviously trained models as the baseline for the diluted model.Deception engine 190 makes the previously trained model a less accurateversion of the baseline truth in the original model. In anotherembodiment, deception engine 190 progressively inserting random wrongresponses. In an example, deception engine 190 returns randomized,incorrect responses from time to time to a user to perturb anystatistical data gathered by the attacker.

In another embodiment, deception engine 190 redirects the user to ahoneypot model. In this example, deception engine 190 uses a model thatmimics the functionality of the original model, but is trained with datathat is loosely representative of the original baseline truth, but issimilar enough to fool an attacker. Deceptive responses by deceptionengine 190 can help invalidate the already extracted information in theattacker. In another example, deception engine 190 changes the fidelityof model responses without altering the original dialogue flow with auser. In this example, deception engine 190 slows or disrupts theinformation accumulation in a hypothetical adversary. In anotherexample, deception engine 190 escalates the conversation to a humanresponder. In this example, deception engine 190, based on the riskscore of the user, initiates a notification to activate a humanresponder to intervene in the conversation.

Probe 195 is a sub program of program 200 that uses hidden dialoguegeneralizations of Completely Automated Public Turing test to tellComputers and Humans Apart (Captchas). Captchas are a type ofchallenge-response test used in computing to determine whether or notthe user is human. In an embodiment, probe 195 sends a probe to a uservia virtual agent program 175, evaluates the probe response and updatesthe risk score for a user in light of the response. In an example, probe195 injects occasional probes based on the risk score of a user. If theuser risk score is borderline between low risk and some risk, then probe195 can intervene in the conversation of virtual agent program 175 andthe user to prove that the user is human. Probe 195 adds data collectedfrom responses and information derived from probe 195 directly to botshield database 182.

In one embodiment, program 200 operates on server 120. In anotherembodiment, program 200 operates on another computer in a server-basedsetting, for example on a server computer not shown. In yet anotherembodiment, program 200 operates on computing device 110 simultaneouslywith server 120 interconnected through network 185. Program 200 providesthe capability to detect and mitigate adversarial conversations withvirtual agents. Program 200 is capable of utilizing Wi-Fi technology,Bluetooth, Near Field Communication tags (NFC), Global System for MobileCommunications (GSM), and Global Positioning System Technology (GPS) tocommunicate with computing device 110.

In an example embodiment, program 200 operates as a code snippet withinone or more applications on computing device 110. Code snippets definethe scope of interactivity between the snippets and the application,(e.g., program 200 hosted by a web browser application on server 120).For example, program 200 is a function within web browser 150, and theprocesses of program 200 occur automatically (i.e., without userintervention) during operation of web browser 150 as initiated byprogram 200. The dynamic code snippet elements provide scriptingsupport. The variables enable dialog between program 200, through server120, graphical user interface 130, web browser 150, and virtual agentprogram 175.

In an embodiment, program 200 is capable of being implemented as anindependent anomaly detection system that is capable of interfacing withthe dialogue system of virtual agent program 175 to provide conversationsecurity. Program 200 detects anomalous and suspicious conversations byleveraging conversational context through preceding model queries by aconversation. In an example, program 200 can operate as a plug-in forvirtual agent as a monitoring capability operating on conversation logs.The anomaly detection subsystem could be used as a stand-alone programto feed an operation dashboard with anomaly monitoring results. In thisexample, deception engine 190 and probe 195 are integrated with orcooperate with the dialogue runtime as deception engine 190 and probe195 are manipulating conversation flow with a user. Each subsystem isextensible and able to learn from the encounters with various attackers.Extensible means that additional detection models are capable of beingimplemented to program 200, additional mitigation responses are capableof being added to program 200, and additional probe choices are capableof being added to program 200.

In another embodiment, program 200 functions as model security tomonitor and detect anomalies at the model application program interface(API) level. Program 200 is able to provide model specific detectionaccording to user or institutional preference.

In an embodiment, program 200 detects a medium-risk value for anencounter between a user and virtual agent program 175. In an example,program 200 determines, utilizing detector 180, deception engine 190,and probe 195, that the user just barely passes the threshold value formitigation by program 200. In this example, program 200 determines whichmitigation procedure is the best response to a possibly suspicious userbased upon consulting bot shield database 182. Program 200 analyzes thehistory of interactions with similar users and weighs the strength andutility of employing a specific probe against the risk of deterring agood user.

FIG. 2 is a flowchart depicting program 200, a program for detecting andmitigating adversarial conversations with virtual agents, in accordancewith an embodiment of the present invention.

In step 210, program 200 determines a risk value for an interaction. Inone embodiment, program 200 analyzes one or more utterances, usingdetector 180, from a user while monitoring the conversation between theuser and a virtual agent. In an example embodiment, program 200intercepts each virtual agent response and extracts the virtual agentresponse log entry. Program 200 uses the virtual agent response logentry as input to invoke an ensemble of “N” anomaly detection models,where each model implements a different anomaly detection strategy. Theindividual detection models in the ensemble run in parallel toindividually compute a risk value, which is merged into an ensemble riskscore using a weighted ensemble function. The ensemble may include oneor more of the following models, as well as other models: a Markov modelto detect unlikely interaction (improbable transitions), a timing modelthat inspects time tags to detect suspiciously fast answers for a humanbeing, a confidence monitoring model that flags the occurrence of lowconfidence scores on recognized intents, and a dialogue progress modelthat identifies the absence or presence of dialogue progress indicators,such as the occurrence of certain target states in the dialogue (e.g.,the phrase “sell something” could raise the risk value score for aninteraction based upon the context of the interaction with virtual agentprogram 175. The Markov model can be built from the dialogue flow graphusing transition frequencies from dialogue states, an informationleakage tracking model to track the distribution of queries to the modelin the model's feature space and determine risky levels of informationexposure (i.e., is the exposed information sufficient to replicate modefunctionality by an adversary).

In an example, program 200 computes a risk value for a user utilizingthe Markov detection module individually. In this example, program 200computes a high-risk value for a user because the dialog log entry tothe virtual agent is characteristic of known attacks. The dialog logentry by the user conforms to the pattern of a known attack and isdeemed intrusive. Program 200 uses the similarity of the log entry toknown attacks to compute a high-risk value to the user.

In an embodiment, program 200 merges the risk scores from the individualdetection models into a single risk score, R, using a weighted ensemblefunction. The weights of the function may be adapted over time. Program200 updates the user risk score in the virtual agent using the mergedrisk value. Program 200 incrementally updates all the anomaly detectionmodels in the ensemble using the dialogue log entry. In an example,program 200 receives a dialog log entry and utilizes a combination oftwo anomaly detection models to compute a risk score for a user. Program200 utilizes the timing anomaly detection model to determine that theresponse time by the user is consistent with the timing patternassociated with a known attacker. Program 200 assigns an individual riskscore, “r1” based on the timing anomaly detection model. Program 200utilizes the dialog progression anomaly detection model to determinethat the dialog log entry to the virtual agent are consistent with adialog progression pattern of a known attacker. Program 200 assigns anindividual risk score, “r2” based on the dialog progression anomalymodel. Program 200 combines risk score “r1” and risk score “r2” tocompute a combined risk score value “R=f(r1, r2)”, where “f” is aweighted ensemble function.

FIG. 3 illustrates an example of program 200 operating to detect andmitigate an attack on a virtual agent, in accordance with an embodimentof the present invention. In this embodiment, program 200 operates onserver 120 and monitors the interactions of virtual nurse 375 withstolen model 310, adversary 315, to permit or allow access toconfidential information trained on sensitive data in domain models 320.In this embodiment, stolen model 310 is confidential diagnosisinformation. Adversary 315 is the user that is initiating and continuingdialog with the virtual nurse. Program 200 is the bot shield defensethat guards the interaction between adversary 315 and the virtual nurse.Domain models 320 are confidential diagnosis information specific to apatient. Virtual nurse 375 is a virtual agent chatbot frontend thatconverses with patients to gather symptom descriptions, includinguploaded images. Virtual nurse 375 uses diagnostic medical models togive a confidential final diagnosis, from domain models 320, response tothe patient.

In an example, as depicted in FIG. 3, program 200 analyzes theutterances entered by adversary 315 to virtual nurse 375. In thisexample, program 200 probes the domain models 320 to extract informationfor the reverse engineering of domain models 320. Program 200 determinesthat stolen model 310 was generated via an extraction attack, and stolenmodel 310 is attempting to extract sensitive information from domainmodels 320 training data. Program 200 employs one or more anomalydetection models previously discussed to confirm that virtual nurse 375is under attack. For example, program 200 uses a dialogue progress modelthat identifies adversary 315 is employing an evasion attack becauseprogram 200 detects input from adversary 315 that is attempting to foolvirtual nurse 375.

In decision step 220, program 200 determines whether the risk valuepasses a threshold. In one embodiment, program 200 determines (from step210) that the risk score has passed a threshold based on the history ofutterances, accessed through bot shield database 182, as compared tocurrently detected utterances. Program 200 determines whether a riskvalue passes a threshold based upon the value computed in step 210 as afunction of the computed risk value that is merged into the ensemblerisk score to generate risk value, “R.”

In an example, with regard to FIG. 3, program 200 (determined in step210) simultaneously operates to provide multi-level detection at theconversation security level between adversary 315 and virtual nurse 375and model security level at the API level between stolen model 310 anddomain models 320. Program 200 assesses an “R” score that exceeds thethreshold as determined by the institution in this example. Domainmodels 320 was trained on sensitive data, and contains confidentialinformation. Program 200 also assigns an “R” score to adversary 315based upon the adversarial inputs, as determined by program 200, inresponse to virtual nurse 375.

In an example, the host for virtual nurse 375 sets an institutional riskscore threshold of 3/10. Virtual nurse 375 guards confidential sensitivehealth information, domain models 320, so the institutional risk scoreis set low relative to an institutional threshold that is not guardingsensitive data. Ten is the highest risk score, and one is the lowestrisk score. Program 200 assigns a high-risk value R, based on theindividual values, R=f(r1, r2)”, where “f” is a weighted ensemblefunction, calculated using the individual anomaly detection models. Inthis example, adversary 315 is determined to present a high risk ofbeing an attacker. In this example, program 200, through historicalinformation from bot shield database 182, the dialog progression modelscore, “r1”, assigned based upon the anomalous context of theconversation. Virtual nurse 375 is trained on health information andhealth care data and adversary 315 is initiating dialog related topersonal information, such as disease and treatment information. Program200 determines that the adversary 315 provides excessively fastresponses to virtual nurse 375's baseline questions, as assessed andcalculated by the timing anomaly detection model “r2.” Program 200calculates a high-risk value “R”=10/10, where ten exceeds theinstitutional threshold, of three, for high-risk interactions.

In step 230, program 200 permits access. More specifically, in responseto determining that the risk score does not pass a threshold (decisionstep 220, “no” branch), program 200 permits access to virtual agentprogram 175 (step 230). In this example, program 200 determines that therisk value, as determined in step 210, does not meet a threshold value“R.” In an example, program 200 analyzes each utterance by a user tovirtual agent program 175 and determine that the responses are by ahuman. Program 200 accesses bot shield database 182 and views a historyof similar utterances that were determined to be human with highconfidence. Based upon the exactness of the utterances and theconsistency of the responses with a history of acceptable responses,program 200 allows a user access to virtual program 175.

In another embodiment, program 200 determines, as a result of step 220,that the risk score of a user is low. In an example, program 200computes a risk score of 1/10 for a user where the institutionalthreshold to initiate action based on the risk presented by the user is5/10. In this example, program 200 permits a user and a virtual agent tocontinue conversation. Program 200 enters into a sentry mode, aftercomputing a low-risk value for a user, while allowing seamless,uninterrupted conversation between a user and a virtual agent. Program200 continues monitoring conversations in sentry mode and is capable ofre-computing the risk score for a user based on further interactionswith the virtual agent. In the event that program 200 raises the riskscore again, based on the context of the conversation, or the appearanceof a historically intrusive pattern of utterances, program 200 iscapable of initiating mitigating actions in response to there-calculated risk score.

In step 240, program 200 deters potential harm. More specifically, inresponse to determining that the risk value does pass a threshold(decision step 220, “yes” branch), program 200 initiates anomalymitigation subsystem (step 240). In an example embodiment, the anomalymitigation subsystem operates to deter potential harm and choose apotential path to deter the potential harm based upon an analysis of thestrength and utility of providing one form of deterrence to a user ascompared to the risk that program 200 may be deterring a “good” user orhuman user.

In one embodiment, program 200 activates mitigating actions, viadeception engine 190 and probe 195, based on the specific risk leveldetermined, and the specific threshold that was passed as a function ofthe determined risk level. Program 200 alters the dialogue flow andre-directs the conversations with a user into a safe area of apreviously determined dialogue tree. Program 200 can adjust the fidelityof responses from virtual nurse 375 with respect to the R value. In anexample, virtual nurse 375 has two fidelity levels for virtual nurse375's response to a user. Virtual nurse 375, through program 200,initiates the original, high fidelity, virtual nurse 375 response to thequestions of a user or virtual nurse 375 initiates a low-fidelityresponse to a user to mitigate the high-risk responses by the user.Program 200 activates a mitigation actions that terminate theconversation by stating “I am not trained on this—for further helpplease call 1-800 . . . ” Program 200 intervenes each time the user riskscore passes the “high risk” threshold, based upon the “R” valueassigned in step 210. Program 200 changes the dialogue flow andredirects virtual nurse 375's response to a previously generated lowfidelity response to mitigate the interaction between adversary 315 andvirtual nurse 375. In another example program 200 delays the response toa user in accordance with the user's response to the probe or thecalculated risk score. In this example, a user's utterances to virtualnurse 375 become increasingly similar with known patterns of high-riskutterances. Program 200 increases the risk score of the userproportionally to each high-risk response. As the risk score getshigher, program 200 introduces a longer delay before sending a responseback to the user.

In another embodiment, program 200 is capable of mitigating anddeterring potential attacks by changing the fidelity of model responsesby virtual nurse 375 without altering original dialogue flow of virtualnurse 375. Program 200 changes the fidelity of model responses byvirtual nurse 375 to slow or disrupt the information accumulation of ahypothetical adversary. Program 200 inserts each protected model insidean ensemble of lower fidelity models. In various embodiments, program200 determines actual model responses by selecting a fidelity modelaccording to the current user risk scores. The higher the user risk, asdetermined in step 210, the lower the response level. In an example,fidelity level, F, is determined as a function of risk score R. Fidelitylevel 1 is the highest level and is consistent with the originalresponse by virtual agent 175 to a human user with a low risk score,fidelity level 2 would be lower, fidelity level 3 is even lower, up tofidelity level N, as determined by institutional or user preferences.

In this another embodiment, program 200 is capable of generating lowerfidelity responses through creating additional models that result inlower fidelity responses to a perceived attacker. Program 200 is capableof using a progressive model dilution. Diluting a model is a method thatuses a previously trained model as the ground truth for the dilutedmodel. As a result, program 200, through deception engine 190, makes thepreviously trained model a less accurate version of the baseline truthin the original model. Program 200 is capable of infinitely chainingeach low-fidelity response to a progressively lower fidelity response.

In an additional example embodiment, with respect to FIG. 3, responsiveto determining that the risk value passes a threshold (decision step220, “yes” branch), program 200 performs mitigating actions in responseto a spam attack. In this example, program 200 determines, based on thesignature response to a threshold high fidelity question presented byvirtual nurse 375. Program 200 assists virtual nurse to validate thatadversary 315 is an attacker by providing increasingly lower fidelityquestions to adversary 315. Program 200 presents adversary 315 with afidelity level 3 question. Based upon adversary 315's completelyunsatisfactory answer to the question presented by program 200, program200 presents adversary 315 with a fidelity level 10 question. Program200 determines that adversary 315 is bot generated spam trafficattempting to drive up operations cost for the virtual agent based onthe incomprehensible responses to the questions presented by program200.

In another embodiment, program 200 is capable of inserting random wrongresponses to deter potential harm, and program 200 is capable ofchanging the tone or manner in which virtual agent program 175 interactswith a user. In an example, program 200 returns random incorrectresponses from time to time to disrupt any statistical data attemptingto be gathered by the attacker. Program 200 can adapt the rate of randomresponses according to the desired fidelity level.

In an additional example embodiment, with respect to FIG. 3, responsiveto determining that the risk value passes a threshold (decision step220, “yes” branch), program 200 performs mitigating actions in responseto a signature proprietary model functionality extraction attack. Inthis example, program 200 determines that adversary 315, based on thecharacter of the questions and responses to virtual nurse 375, isattempting to harvest proprietary model functionality information.Adversary 315 presents virtual nurse 375 with a rapid succession ofquestions that are related to an entire area of a decision tree ofresponses by virtual nurse 375. Adversary 315 presents a series ofanswers to virtual nurse 375 that have a “yes” response. Adversary 315then presents the same series of answers to the same questions posed byvirtual nurse 375 with “no” responses. Program 200 recognizes thesignature attack method and assists virtual nurse 375 to randomlyprovide adversary 315 with random unrelated responses to preventadversary 315 from gathering proprietary model functionality of virtualnurse 375.

In another embodiment, program 200 is capable of redirecting a user to ahoneypot model. In an example, program 200 creates and uses a “honeypot”to mimic the functionality of the original model, but is trained withdata that is loosely representative of the original ground truth, butclose enough to fool an attacker. With respect to FIG. 3, program 200 iscapable of generating a virtual “honeypot” that is similar to domainmodel 320. In this example, an attacker would perform an extractionattack through reverse engineering the “honeypot” thus be deterred fromactually capturing the information in actual domain model 320. In thisexample, stolen model 310 would be a copy of a deception, i.e., the“honeypot” model.

In an additional example embodiment, with respect to FIG. 3, responsiveto determining that the risk value passes a threshold (decision step220, “yes” branch), program 200 performs mitigating actions in responseto an attack by adversary 315 to extract proprietary information fromtraining data of virtual nurse 375. In this example, program 200determines that adversary 315 is attempting to extract domain model 320.Program 200 determines, based upon the signature questions thatadversary 315 presents to virtual nurse 375 to circumvent virtual nurse375's security protocols, program 200, through deception engine 190,adapts virtual nurse 375 to recreate virtual nurse 375 as ahigh-interaction honeypot. Virtual nurse 375, now functioning as ahigh-interaction honeypot, gathers in-depth information about adversary315's tools and techniques used to extract information. Program 200presents adversary 315 with a real system, virtual nurse 375 temporarilyrepurposed as a honeypot model, gives adversary 315 root privileges ofthe virtual nurse system, and allows adversary 315 access to thehoneypot system. Program 200 gathers detailed information aboutadversary 315's extraction attacks, develops a signature for theextraction attack and stores adversary 315's profile, and methods ofattack in bot shield database 182.

In another embodiment, program 200 uses probe 195 to quickly develop arisk score for a user or to further analyze an indeterminate user andassign a risk score as a function of the further analysis by probe 195.Program 200 ranks available probes by plausibility to occur in a normalconversation. Program 200 injects occasional probes based on the currentrisk score of a user and program 200 can adjust the strength and needfor a probe, and the frequency of injection based on the current riskscore of a user. Program 200 evaluates the response by a user to a probeand updates the risk score accordingly, or program 200 can employfurther probes. Program 200 adds information to bot shield database 182as a function of the information provided by probe 195. Program 200,through probe 195, sends out a request or other form of inquiry to getmore information back from the user in response to the probe. Program200, through probe 195, is capable of intervening in a conversationbetween a user and virtual nurse 375 to further assess a risk score asassigned to the user. In an example, program 200 interjects into aconversation and directly requests that the user to prove that the useris a human through one or more probes, such as a “Captchas.”

In an example, program 200 employs probe 195 in response to a new user,or a user with no signature history of responses stored in bot shielddatabase 182. Probe 195 may be engaged at a low frequency by program200. In another example, program 200 employs a probe to verbalutterances by a user and inserts phrases such as, “I'm sorry I am nottrained on this, could you please rephrase,” “Did you mean X?” (where Xis something that virtual agent program 175 is highly confident that Xis not what the previous user utterance was related to, i.e., negativeconfirmation), or program 200 can utilize probe 195 to employ asuperfluous question that requires more than a “yes” or “no” answer andthat relates to the current context. (e.g., “when did you first acquireyour car” in a conversation about car insurance).

In an additional example embodiment, with respect to FIG. 3, responsiveto determining that the risk value passes a threshold (decision step220, “yes” branch), program 200 performs mitigating actions in responseto a poisoning attack. In this example, virtual nurse 375 continuouslylearns through production use. The more interaction that virtual nurse375 has with “good” users the better that virtual nurse 375 functionsand the more virtual nurse 375 evolves. In this example, program 200determines, based on the utterances by adversary 315, that adversary 315is altering virtual nurse 375's training data. Program 200 determinesthat the topics of conversation imitated, and continued, by adversary315 are driving the conversation, originally resigned to health caretopics, off course into unrelated topic areas. In response, program 200directs virtual nurse 375 into a “safe mode.” Virtual nurse 375 does notreveal any confidential, sensitive, or proprietary information,including any proprietary model functionality, such as domain model 320to adversary 315. In the event that adversary 315 continues to attemptto “poison” virtual nurse 375, program 200 terminates the connection,discontinuing the conversation, between virtual nurse 375 and adversary315.

In an additional example embodiment, with respect to FIG. 3, responsiveto determining that the risk value passes a threshold (decision step220, “yes” branch), program 200 performs mitigating actions by combiningthe mitigation actions of decreasing fidelity in response to a user,giving wrong responses to a user, using the honeypot deception method,and using one or more probes to update a user's risk score.

FIG. 4 depicts a block diagram of components of server 120, inaccordance with an illustrative embodiment of the present invention. Itshould be appreciated that FIG. 4 provides only an illustration of oneimplementation and does not imply any limitations with regard to theenvironments in which different embodiments may be implemented. Manymodifications to the depicted environment may be made.

Server 120 includes communications fabric 402, which providescommunications between cache 416, memory 406, persistent storage 408,communications unit 410, and input/output (I/O) interface(s) 412.Communications fabric 402 can be implemented with any architecturedesigned for passing data and/or control information between processors(such as microprocessors, communications and network processors, etc.),system memory, peripheral devices, and any other hardware componentswithin a system. For example, communications fabric 402 can beimplemented with one or more buses or a crossbar switch.

Memory 406 and persistent storage 408 are computer readable storagemedia. In this embodiment, memory 406 includes random access memory(RAM). In general, memory 406 can include any suitable volatile ornon-volatile computer readable storage media. Cache 416 is a fast memorythat enhances the performance of computer processor(s) 404 by holdingrecently accessed data, and data near accessed data, from memory 406.

Program 200 may be stored in persistent storage 408 and in memory 406for execution by one or more of the respective computer processors 404via cache 416. In an embodiment, persistent storage 408 includes amagnetic hard disk drive. Alternatively, or in addition to a magnetichard disk drive, persistent storage 408 can include a solid state harddrive, a semiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 408 may also be removable. Forexample, a removable hard drive may be used for persistent storage 408.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage408.

Communications unit 410, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 410 includes one or more network interface cards.Communications unit 410 may provide communications through the use ofeither or both physical and wireless communications links. program 200may be downloaded to persistent storage 408 through communications unit410.

I/O interface(s) 412 allows for input and output of data with otherdevices that may be connected to server 120. For example, I/O interface412 may provide a connection to external devices 418 such as a keyboard,keypad, a touch screen, and/or some other suitable input device.External devices 418 can also include portable computer readable storagemedia such as, for example, thumb drives, portable optical or magneticdisks, and memory cards. Software and data used to practice embodimentsof the present invention, e.g., program 200, can be stored on suchportable computer readable storage media and can be loaded ontopersistent storage 408 via I/O interface(s) 412. I/O interface(s) 412also connect to a display 420. Display 420 provides a mechanism todisplay data to a user and may be, for example, a computer monitor.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

What is claimed is:
 1. A method for detecting and mitigating adversarialvirtual interactions, the method comprising: detecting, by one or moreprocessors, a user communication that is interacting with a virtualagent; determining, by one or more processors, a risk level associatedwith the detected user communication based on one or more actionsperformed by the detected user while interacting with the virtual agent;in response to determining that the determined risk level associatedwith the detected user communication exceeds a risk level threshold,initiating, by one or more processors, a mitigation protocol oninteractions between the detected user and the virtual agent, whereinthe mitigation protocol is based on the actions performed by thedetected user while interacting with the virtual agent; and in responseto initiating the mitigation protocol on interactions between thedetected user and the virtual agent, generating, by one or moreprocessors, a lower fidelity response from the virtual agent to theuser, wherein the lower fidelity response is a progressive dilution ofthe precision of language of an original response from the virtual agentto the user prior to the detected user exceeding the risk levelthreshold.
 2. The method of claim 1, wherein initiating the mitigationprotocol further comprises: altering, by one or more processors,communication responses from the virtual agent to the user; anddirecting, by one or more processors, dialogue between the user and thevirtual agent into a pre-determined dialogue tree, wherein thepre-determined dialogue tree is a response protocol that concealsconfidential data associated with the virtual agent.
 3. The method ofclaim 1, wherein initiating the mitigation protocol further comprises:terminating, by one or more processors, interactions between the userand the virtual agent; reporting, by one or more processors, data of theinteractions to one or more enterprise network databases; and storing,by one or more processors, data of the interactions, wherein the data ofthe dialogue may include one or more frequently used terms, andtechniques designed to extract information from the virtual agent. 4.The method of claim 1, wherein initiating the mitigation protocolfurther comprises: determining, by one or more processors, that the risklevel associated with the user is increasing; and delaying, by one ormore processors, a time period of the response from the virtual agent tothe user in proportion to the increasing risk level associated with theuser.
 5. The method of claim 1, wherein initiating the mitigationprotocol further comprises: identifying, by one or more processors, asignature for the user, wherein the signature is associated with anextraction attack on the virtual agent; generating, by one or moreprocessors, a decoy model of confidential information associated with acommunication by the user wherein formatting of the decoy model ofconfidential information matches formatting of an actual instance ofconfidential information; presenting, by one or more processors, thedecoy model to the user; and extracting, by one or more processors, datafrom the user, wherein the data extracted from the user includesprocedures used by the user to extract proprietary data.
 6. The methodof claim 1, wherein determining the risk level associated with thedetected user communication based on one or more actions performed bythe detected user while interacting with the virtual agent furthercomprises: activating, by one or more processors, a probe to retrievemore information from the user; and updating, by one or more processors,the risk level associated with the user communication based on theinformation from the probe.
 7. The method of claim 1, furthercomprising: responsive to determining that the determined risk levelassociated with the detected user communication exceeds the risk levelthreshold, initiating, by one or more processors, a plurality ofmitigation actions in combination, wherein the mitigation actions areselected from the group consisting of: generating a lower fidelityresponse from the virtual agent to the user, terminating the interactionbetween the user and the virtual agent, activating a probe to retrieveinformation from the user, delaying a time period of the response fromthe virtual agent to the user, and generating a decoy model ofconfidential information.
 8. The method of claim 1, further comprises:delaying, by one or more processors, a time period of the response fromthe virtual agent to the user based on a determined mitigation protocol;and generating, by one or more processors, concurrently the determinedmitigation protocol based on one or more conversations with the user. 9.A computer program product for detecting and mitigating adversarialvirtual interactions, the computer program product comprising: one ormore computer readable tangible storage media and program instructionsstored on at least one of the one or more computer readable storagemedia, the program instructions readable/executable by one or morecomputer processors and further comprising: program instructions todetect a user communication that is interacting with a virtual agent;program instructions to determine a risk level associated with thedetected user communication based on one or more actions performed bythe detected user while interacting with the virtual agent; programinstructions to in response to determining that the determined risklevel associated with the detected user communication exceeds a risklevel threshold, initiate, a mitigation protocol on interactions betweenthe detected user and the virtual agent, wherein the mitigation protocolis based on the actions performed by the detected user while interactingwith the virtual agent; and program instructions to in response toinitiating the mitigation protocol on Interactions between the detecteduser and the virtual agent, generating, by one or more processors, alower fidelity response from the virtual agent to the user, wherein thelower fidelity response is a progressive dilution of the precision oflanguage of an original response from the virtual agent to the userprior to the detected user exceeding the risk level threshold.
 10. Thecomputer program product of claim 9, wherein initiating the mitigationprotocol further comprises program instructions, stored on the one ormore computer readable storage media, which when executed by aprocessor, cause the processor to: alter communication responses fromthe virtual agent to the user; and direct dialogue between the user andthe virtual agent into a pre-determined dialogue tree, wherein thepre-determined dialogue tree is a response protocol that concealsconfidential data associated with the virtual agent.
 11. The computerprogram product of claim 9, wherein initiating the mitigation protocolfurther comprises program instructions, stored on the one or morecomputer readable storage media, which when executed by a processor,cause the processor to: terminate interactions between the user and thevirtual agent; report data of the interactions to one or more enterprisenetwork databases; and store data of the interactions, wherein the dataof the dialogue may include one or more frequently used terms, andtechniques designed to extract information from the virtual agent. 12.The computer program product of claim 9, wherein initiating themitigation protocol further comprises program instructions, stored onthe one or more computer readable storage media, which when executed bya processor, cause the processor to: determine that the risk levelassociated with the user is increasing; and delay a time period of theresponse from the virtual agent to the user in proportion to theincreasing risk level associated with the user.
 13. The computer programproduct of claim 9, wherein initiating the mitigation protocol furthercomprises program instructions, stored on the one or more computerreadable storage media, which when executed by a processor, cause theprocessor to: identify a signature for the user, wherein the signatureis associated with an extraction attack on the virtual agent; generate adecoy model of confidential information associated with a communicationby the user wherein formatting of the decoy model of confidentialinformation matches formatting of an actual instance of confidentialinformation; present the decoy model to the user; and extract data fromthe user, wherein the data extracted from the user includes proceduresused by the user to extract proprietary data.
 14. The computer programproduct of claim 9, wherein determining the risk level associated withthe detected user communication based on one or more actions performedby the detected user while interacting with the virtual agent furthercomprises program instructions, stored on the one or more computerreadable storage media, which when executed by a processor, cause theprocessor to: activating a probe to retrieve more information from theuser; and update the risk level associated with the user communicationbased on the information from the probe.
 15. The computer programproduct of claim 9 stored on the one or more computer readable storagemedia, which when executed by a processor, cause the processor to:responsive to determining that the determined risk level associated withthe detected user communication exceeds the risk level threshold,initiate a plurality of mitigation actions in combination, wherein themitigation actions are selected from the group consisting of: generatinga lower fidelity response from the virtual agent to the user,terminating the interaction between the user and the virtual agent,activating a probe to retrieve information from the user, delaying atime period of the response from the virtual agent to the user, andgenerating a decoy model of confidential information.
 16. The computerprogram product of claim 9, stored on the one or more computer readablestorage media, which when executed by a processor, cause the processorto: delay a time period of the response from the virtual agent to theuser based on a determined mitigation protocol; and generateconcurrently the determined mitigation protocol based on one or moreconversations with the user.
 17. A computer system for detecting andmitigating adversarial virtual interactions, the computer systemcomprising: one or more computer processors; one or more computerreadable storage media; and program instructions stored on the one ormore computer readable storage media for execution by at least one ofthe one or more computer processors, the program instructionscomprising: program instructions to detect a user communication that isinteracting with a virtual agent; program instructions to determine arisk level associated with the detected user communication based on oneor more actions performed by the detected user while interacting withthe virtual agent; program instructions to in response to determiningthat the determined risk level associated with the detected usercommunication exceeds a risk level threshold, initiate, a mitigationprotocol on interactions between the detected user and the virtualagent, wherein the mitigation protocol is based on the actions performedby the detected user while interacting with the virtual agent; andprogram instructions to in response to initiating the mitigationprotocol on interactions between the detected user and the virtualagent, generating, by one or more processors, a lower fidelity responsefrom the virtual agent to the user, wherein the lower fidelity responseis a progressive dilution of the precision of language of an originalresponse from the virtual agent to the user prior to the detected userexceeding the risk level threshold.
 18. The computer system of claim 17wherein initiating the mitigation protocol further comprises programinstructions, stored on the one or more computer readable storage media,which when executed by a processor, cause the processor to: altercommunication responses from the virtual agent to the user; and directdialogue between the user and the virtual agent into a pre-determineddialogue tree, wherein the pre-determined dialogue tree is a responseprotocol that conceals confidential data associated with the virtualagent.
 19. The computer system of claim 17 wherein initiating themitigation protocol further comprises program instructions, stored onthe one or more computer readable storage media, which when executed bya processor, cause the processor to: terminate interactions between theuser and the virtual agent; report data of the interactions to one ormore enterprise network databases; and store data of the interactions,wherein the data of the dialogue may include one or more frequently usedterms, and techniques designed to extract information from the virtualagent.
 20. The computer system of claim 17 wherein initiating themitigation protocol further comprises program instructions, stored onthe one or more computer readable storage media, which when executed bya processor, cause the processor to: determine that the risk levelassociated with the user is increasing; and delay a time period of theresponse from the virtual agent to the user in proportion to theincreasing risk level associated with the user.
 21. The computer systemof claim 17 wherein initiating the mitigation protocol further comprisesprogram instructions, stored on the one or more computer readablestorage media, which when executed by a processor, cause the processorto: identify a signature for the user, wherein the signature isassociated with an extraction attack on the virtual agent; generate adecoy model of confidential information associated with a communicationby the user wherein formatting of the decoy mod& of confidentialinformation matches formatting of an actual instance of confidentialinformation; present the decoy model to the user; and extract data fromthe user, wherein the data extracted from the user includes proceduresused by the user to extract proprietary data.
 22. The computer system ofclaim 17 wherein determining the risk level associated with the detecteduser communication based on one or more actions performed by thedetected user while interacting with the virtual agent further comprisesprogram instructions, stored on the one or more computer readablestorage media, which when executed by a processor, cause the processorto: activating a probe to retrieve more information from the user; andupdate the risk level associated with the user communication based onthe information from the probe.
 23. The computer system of claim 17stored on the one or more computer readable storage media, which whenexecuted by a processor, cause the processor to: responsive todetermining that the determined risk level associated with the detecteduser communication exceeds the risk level threshold, initiate aplurality of mitigation actions in combination, wherein the mitigationactions are selected from the group consisting of: generating a lowerfidelity response from the virtual agent to the user, terminating theinteraction between the user and the virtual agent, activating a probeto retrieve information from the user, delaying a time period of theresponse from the virtual agent to the user, and generating a decoymodel of confidential information.
 24. The computer system of claim 17further comprises program instructions, stored on the one or morecomputer readable storage media, which when executed by a processor,cause the processor to: delay a time period of the response from thevirtual agent to the user based on a determined mitigation protocol; andgenerate concurrently the determined mitigation protocol based on one ormore conversations with the user.